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ABSTRACT 
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effective tool must incorporate criteria from the discipline of library and 
information science, and also from the disciplines of graphic design and 
linguistics. This study identifies, defines, and discusses the 
characteristics of a valuable web site. A series of evaluation forms organize 
and display critical criteria taken from the current literature using the 
methodologies of content analysis and feature analysis. Prototype testing was 
conducted on a selective sample of 30 Internet web sites. Modifications and 
refinements were made to the instrument and incorporated into an accompanying 
user's manual. This manual defines and discusses the benchmark criteria and 
explains proper implementation of the instrument in any adult reference 
setting. No single element such as graphics, content, or user interface 
determines the quality of an Internet web site. The evaluation instrument 
affords reviewers the opportunity to examine multimedia, content, and 
user- interface while providing a quantitative means of rating each respective 
section. The instrument was created to evaluate individual web sites, not 
search engines or catalog sites. Appendices comprising half the document 
consist of the evaluation instrument and User's Guide. (Contains 29 
references.) (Author/SWC) 



****************************************************** * * ******* ************ 

♦ Reproductions supplied by EDRS are the best that can be made 

♦ from the original document. 



ED 413 899 



AN EVALUATION INSTRUMENT 
FOR INTERNET WEB SITES 



A Master’s Research Paper submitted to the 
Kent State University School of Library Science 
in partial fulfillment of the requirements 
for the degree Master of Library Science 



"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 




Stephanie Plank Livengood 



by 



R. Du Mont 




originating it. 



July, 1997 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)." 



2 



bsst cofi 



ABSTRACT 



The purpose of this research project was to develop a comprehensive evaluation tool for 
reference librarians in adult service divisions to use in selecting World Wide Web sites as reference 
sources. Search engine and catalog sites are not included. Traditional evaluation criteria, endorsed 
and applied by librarians over the years, are not sufficient for the evaluation of today’s hypermedia 
web site environment. An effective tool must incorporate criteria, not only from the discipline of 
library and information science, but also from the disciplines of graphic design and linguistics. 
Therefore, this study identified, defined, and discussed the characteristics of a valuable web site. The 
end product, a series of evaluation forms, organizes and displays critical criteria taken from the 
current literature using the methodologies of content analysis and feature analysis. Prototype testing 
was conducted on a selective sample of thirty Internet web sites. Modifications and refinements were 
made to the instrument and incorporated into an accompanying manual entitled “User’s Guide.” 
This guide defines and discusses the benchmark criteria and explains proper implementation of the 
instrument in any adult reference setting. 
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CHAPTER I 



INTRODUCTION 

Electronic access to information is undoubtedly changing the way reference librarians 
find, select, and disseminate information. An example of this is seen in the vast number of 
sources on the Internet which are made accessible via the World Wide Web (WWW). The 
most recent figures exemplifying the exponential growth rate of the Internet are mind- 
boggling. The total number of registered sites as of 1997 has reached the staggering figure 
of 9.4 million, and the number of individual web pages is estimated to be somewhere 
between thirty and fifty million.' One explanation for these extremely high figures is the 
ease of use, standardization, and portability of hypertext markup language, HTML, which 
allows practically anyone to create web pages and web sites. 

Internet web sites present reference librarians with a multitude of unprecedented 
challenges. They are yet another source with which librarians must be familiar and consider 
for an information retrieval process. Their placement within the Internet can be described 
as chaotic - a characteristic in stark contrast to the typical libraries in which order and 
organization serve as the fundamental keystone. The quality of information on World Wide 
Web sites varies tremendously; some sites are excellent while others are extremely poor. 
Finally, the increasing complexity requires that they be put through the rigors of an 
evaluation and review process before they can ever be selected as reference sources. 
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Today’s reference librarians and information scientists are in the prime position to 
embrace these challenges and, at the same time, to advance the profession. As the Internet 
becomes more commonplace in the American home, the knowledge and skills of librarians 
in information acquisition, selection, and evaluation will become even more valuable. 
Internet users will need to know how to use complex search engines and indexing systems 
present at web sites. More importantly, as the number of sites increases and sources overlap 
with one another, users will need to know which sites to access for quality information. 

Reference service is indeed entering a new era as librarians prepare to provide a more 
sophisticated electronic reference service in the next century. An Internet reference and 
review service will be necessary to meet the swiftly changing needs of patrons. The basis 
of such a program is indeed nothing new to libraries. They have been offering a similar 
service to patrons for many years who have come to trust and rely on their recommendations 
for print, audio, and video materials. An Internet reference and review program will require 
much time and funding, although the service should be regarded as a sound investment of 
public monies in the future. 

At present, librarians currently have only two options from which to choose in 
locating quality reviews of academic and general reference web sites. They may consult any 
of the four evaluating services provided by library and information scientists listed below, 
or they may perform their own individual evaluations and reviews. The first source. College 
& Research Libraries News, publishes a monthly column entitled “Internet Reviews,” edited 
by Sara Amato of Central Washington University. She critiques three or four sites each 
month that are most appropriate to the college library setting. Also, the official publication 
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of the American Library Association, Library Journal , contains a monthly column entitled 
“Web Watch” by Boyd Collins. He evaluates three or four sites per month which would be 
appropriate to public libraries. A third reviewing source, the official web site of the H. W. 
Wilson Company, http://www.hwwilson.com, offers a column, “Rettig on Reference,” by 
James Rettig of William and Mary College. Each month he discusses and reviews a variety 
of Internet resources, including several World Wide Web sites. The final reviewing service, 
the Infofilter Project, is a group effort by library and information scientists nationwide to 
provide in-depth description, evaluation, and review of web sites. The Project is accessible 
through the University of Southern California’s web site at http://www.usc.edu/users/help/ 
flick/Infofilter/. This professional group has reviewed approximately forty-two web sites to 
date.^ 

The total number of reviews published by these four evaluating services amounts to 
less than one hundred. This distressing figure alone is reason enough for librarians to 
conduct their own evaluations and reviews, not to mention the fact that librarians do indeed 
have their own valuable reviewing skills to offer. A reference and review service not only 
would benefit patrons, but also would be an additional public resource the library could offer 
with confidence and understanding. Furthermore, if librarians are to be experts at managing 
Internet information, as they are at other information sources, they must now take the lead 
in the evaluation and review process of Internet web sites. 

A review of the literature reveals a significant number of electronic publications on 
the topic, “Evaluating Internet Resources.” Contributing authors are confronting this issue 
head on. The primary theme throughout these articles is the call for the application of critical 
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thinking to the evaluation process of Internet information. The major difference of opinion 
among the authors is their proposed methodology for the implementation of this process. An 
example of a technology-based approach is proposed by Matthew T. Ciolek of the Australian 
National University, Canberra. He calls for an evaluative procedure that is “...simple, ... 
automated, ... and carried out by a piece of software.”^ On the other hand, Louis B. 
Rosenfeld, instructor of Library and Information Science at the University of Michigan, 
contends that “... no automated tool can assess the quality of information resources on the 
Internet; intrinsic issues of format, content, context, source, location, authority, cannot be 
judged by software. Instead, quality assessment continues to be almost wholly the product 
of intellectual labor.”"* 

The purpose of this research project was to develop such an intellectual tool for adult 
service reference librarians to use in selecting Internet web sites as reference sources. Search 
engines and catalog sites are not included because of time and complexity constraints. 
Traditional evaluation criteria, endorsed and applied by librarians over the years, are not 
sufficient for the evaluation of today’s hypermedia web site environment. An effective, 
evaluation instrument must incorporate a taxonomy based on criteria taken not only from the 
discipline of library and information science, but also from the disciplines of graphic design 
and linguistics, among others. 

Development of this instrument was accomplished in four phases. Part I focused on 
the identification, definition, and discussion of web site evaluation criteria taken from current 
literature in the above-mentioned disciplines. The second part of the study involved 
transposing these criteria onto a series of evaluation forms as presented in Appendix A. In 
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Part III of the study, the prototype was tested on thirty web sites. Quantitative and qualitative 
data documenting the tool’s effectiveness were compiled and reported in Chapter 5 entitled 
“Analysis of Data.” The study came to fruition in Part IV, the follow-up phase, during which 
modification and refinement were made to the instrument. Final commentary regarding the 
worthiness of this study was included in Chapter 6 entitled “Summary and Conclusions.” 
The extensive number of evaluation criteria revealed in the literature review posed 
one of the main limitations of the study. For reasons relating to the functionality of the 
instrument, the researcher selected criteria based on importance, relevance, practicality, and 
testability. A second limitation, the restricted time frame of this project, required placing 
a ceiling on the sample size, making it seem minute in comparison to the vast number of web 
sites comprising the study’s population. 
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CHAPTER II 



REVIEW OF THE LITERATURE 

A literature review on the topic “Evaluating Internet Information” revealed relevant 
and timely materials published since 1993 in the form of monographs, journal articles, and 
electronic publications. The initial search for monographs was performed on the OHIOLINK 
and CLEVENET online library catalogs using the following subject and word search strings: 
“Internet and Librar* and “Internet and Evaluat*.” This same search strategy was repeated 
substituting the term World Wide Web and the acronym, “WWW,” for “Internet.” Results 
of these searches were successful in locating materials providing a broad overview of the 
topic. Two comprehensive Internet guides, consulted during the research phase of this 
project, were the World Wide Web 1996 Unleashed by authors, John December and Neil 
Randall, and The Whole Internet by Ed Krol. A third ancillary source. Librarians on the 
Internet , edited by Robin Kinder, served as a reference source for Internet service issues 
currently being addressed by library and information scientists. 

Additional subject searching for monographs was conducted using the search string, 
“web site and design.” This search revealed numerous publications applicable to all sections 
of the instrument, specifically Part II, the Multimedia Feature Analysis. The monographs 
retrieved were: Designing Large-Scale Web Sites by Darrell Sand, World Wide Web 

Design Guide by Stephen Wilson, and Building the Service-Based Library Web Site by 
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Kristen L. Garlock and Sherry Piontek. The first two publications, written by professional 
graphic artists, provided extensive detail on web site design techniques recognized 
throughout the industry for the presentation of text, graphics, audio and video. These 
guidelines served as a model upon which the benchmark criteria and evaluation questions 
were based. Also, these monographs contained definitions of technical terms pertinent to 
this study that were condensed and incorporated into the glossaries found in Appendix B. 
The latter publication. Building the Service-Based Library Web Site, included many of the 
key design elements cited in the above-mentioned graphic design books, and offered 
additional ideas based on the authors’ experience and expertise as librarians. Some of their 
most useful suggestions incorporated into this study focused on content, accuracy, 
objectivity, and design techniques to benefit the disabled. 

A third group of monographs critical to the human computer interaction and user 
interface issues addressed in this study were: Human-Computer Interaction by professors 
Alan Dix, Janet Finlay, Gregory Abowd, and Russell Beale; Principles and Guidelines in 
Software User Interface Design by Deborah J. Mayhew; and Evaluating Usability of Human- 
Computer Interfaces by Susannah Ravden and Graham Johnson. The textbook. Human 
Computer Interaction, provided essential background information on this discipline, 
commonly referred to as HCI, and offered substantial detail regarding input-output channels, 
e.g., vision, hearing, and touch, through which humans are capable of interacting with 
computers. Chapter 1 5, entitled “Multi-sensory Systems,” addressed specific complexities 
unique to the dynamic Internet environment and many of the challenges it presents to both 
designers and users. Each chapter concluded with a “Recommended Reading” section which 
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led to several other publications cited in the bibliography to this research. The other 
publications by Mayhew and Ravden and Johnson, were most helpful in preparing the user- 
interface questions of Part IV and some multimedia questions in Part II. Their scientifically 
tested criteria and evaluation questions were adapted for incorporation in this tool. 

A literature search of journal articles published since 1993 was accomplished by 
using Library Literature and the CONSORT Library Journal Index at the College of Wooster. 
Relevant articles on the subjects “Evaluating Internet Resources” and “World Wide Web 
Evaluation” were located in leading library and information science journals such as: 
Computers in Libraries . Library Journal . Reference Services Review . Online , and Internet 
World . The most substantive and useful articles retrieved were “Beyond ‘Cool,’: Analog 
Models for Reviewing Digital Resources,” and “Putting the Squeezed on the Information 
Firehose: the Need for Neteditors and Netreviewers” by James Rettig . These two essays not 
only supported the development of a comprehensive web site evaluation instrument for 
librarians, but also listed specific reviewing criteria for Internet publications. Another 
journal article that aided in determining categorical headings for these criteria was “Web 
Watch” published in Library Journal on February 1st, 1996. In this premier column, Boyd 
Collins revealed his original taxonomy for the evaluation of web sites and described some 
accompanying criteria. 

The most abundant source of information on this topic was found on the Internet 
itself A simple boolean search combining the terms, “Internet and Evaluation,” at the Lycos 
site revealed several bibliographies on the subject. The most exhaustive list has been 
compiled by Nicole Auer, a Library Instruction Coordinator at the Virginia Polytechnic 
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Institute and State University. The address of this web site is; http://refserver.lib.vt.edu/lib 
inst/critTHINK.HTM. 

Auer cites over twenty-five Internet publications, thirty print sources, and eight 
professional listservs, created by librarians, information scientists, and researchers. Of these 
Internet articles, approximately twenty were reviewed and analyzed for the purpose of this 
research. Those which served as a base for this project included: “Review of the Five 
Traditional Print Evaluation Criteria” by Jan Alexander and Marsha Tate; “Thinking 
Critically about World Wide Web Resources” by Esther Grassian; “Evaluation of 
Information” by Lisa Janicke Hinchliffe; “Library Selection Criteria for WWW Resources” 
by Carolyn Caywood; “Evaluating Quality on the Net” by Hope N. Tillman; and “Criteria 
for Evaluation of Internet Information Resources” by Alastair Smith. 

Many of these authors re-examined traditional print evaluation criteria taken from 
authoritative sources such as Selection and Evaluation of Reference Sources by Bopp and 
Smith and “Evaluating Reference Books in Theory and Practice” by Norman Stevens, and 
discussed their applicability to the digital medium. All concurred that the majority of these 
criteria were indeed still appropriate, but would, however, require some modification and 
adaptation. Authors Grassian, Caywood, and Smith proposed additional criteria for links, 
structure, access, and multimedia design. Although these areas were neither discussed at 
great length nor described in terms more precise than those such as “user friendliness” and 
“workability,” they were extremely useful in serving as a general outline for the instrument’s 
design. 



CHAPTER III 



RESEARCH OBJECTIVES 

The primary objective of this library science research project was to develop a 
comprehensive, evaluation instrument for adult service reference librarians to use in selecting 
Internet web sites as reference sources (search engine and catalog sites not included). In 
order to design a tool that would address the revolutionary technological advancements in 
today’s hypermedia web site environment, the disciplines of library and information science, 
computer graphic design, and linguistics were examined to determine the most appropriate 
evaluation and review criteria. 

The secondary objective of this project was to develop a tool that was pragmatic and 
easy to use in any library setting. This goal was accomplished by devising a consistent, 
concise format and arrangement. The instrument was divided into four distinctive sections 
which were further divided into subdivisions and arranged by their respective features and 
criteria type. The main elements included in the Multimedia Feature Analysis were graphics, 
video, animation, and audio. The purpose of identifying and examining these features was 
to determine whether or not they enhanced the information content of the site and if so, to 
determine the manner by which this was achieved. Library science and linguistics criteria 
included in the Content Analysis and Evaluation section were scope, authority, accuracy, 
objectivity, organization, and relation to other works. Accompanying evaluation questions 
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in this section focused attention on issues related to quality of the content. The final section, 
User Interface Evaluation, included criteria for navigation, searching, visual clarity, and 
informative feedback and support. By identifying and evaluating these features, the reviewer 
was better able to understand how information is accessed and whether or not it is easy to 
retrieve. 

The third objective, to identify and define the characteristics of a valuable web site, 
was met by developing an instruction manual or “User’s Guide” to be used in concert with 
the instrument. (See Appendix B) The primary purpose of this guide is to explain proper 
implementation of the tool. Definitions of criteria, a discussion of their significance, plus 
suggestions for locating these criteria within the web site are included, also. 

The final objective, to render a tool that was efficacious in yielding desired results, 
was achieved by testing the instrument on a total of thirty Internet sites. Results in the form 
of quantitative and qualitative data exposed strengths and weaknesses of the tool. 
Modifications and adjustments were made in the final phase of this research to produce an 
end product consisting of a qualitative review in the form of a written recommendation to 
accept or reject the web site and a quantitative analysis or numerical rating to be used in 
support of the evaluator’s approval or rejection of the web site. The final version of this 
instrument is foimd in Appendix A. 
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CHAPTER IV 



METHODOLOGY 



Development of the web site evaluation instrument was accomplished by using the 
four-part study described below: Part I - Research Phase; Part II - Design Phase; Part III - 
Testing Phase; and Part IV - Data Analysis Phase. Part I, the Research Phase, entailed the 
identification, definition, and discussion of valuable web site characteristics revealed through 
the literature review. The most critical of these were selected as benchmark criteria based 
on importance, relevance, practicality, and testability. 

During the Design Phase, Part II, the criteria were transposed onto evaluation forms 
which served as the instrument for gathering data. The instrument was divided into four 
main parts preceded by an introductory cover sheet. These main sections were entitled: Part 
I - Technical Description; Part II - Multimedia Feature Analysis; Part III - Content Analysis 
and Evaluation; and Part IV - User Interface Evaluation. Content analysis and feature 
analysis methodologies were employed to allow the reviewer to work through the evaluation 
forms methodically and efficiently. The amount of time to complete a single evaluation was 
originally estimated to be twenty minutes. Results of the testing phase, however, indicated 
a mean time frame of 37 minutes. 

The sequence of the instrument’s individual sections was purposely devised to allow 
the reviewer full benefit from the review session. For example, upon first accessing a web 
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site, the reviewer should record the descriptive information presented at the homepage of the 
site in Part I. During Part II, the Multimedia Feature Analysis, the reviewer would be 
afforded the opportunity to become familiar with the site simply by maneuvering throughout 
to indicate the presence or absence of features, e.g., external and / or internal audio clips, 
black and / or white graphics, and continuous and / or user-activated animation. By Part III, 
Content Analysis and Evaluation, the reviewer should now be prepared to answer specific 
questions regarding content. It was anticipated that by concluding with Part IV, the reviewer, 
having already spent at least fifteen to twenty minutes navigating the site, would be able to 
answer questions regarding navigation, visual clarity, and informative feedback more 
efficiently. 

Part III of this study, the Testing Phase, took place over an eleven day period from 
July 1st through July 1 1th, 1997. During this time, the prototype was tested on a total of 
thirty Internet web sites. A variety of sites comprising the selective sample was chosen based 
upon their potential as reference sources for a public library setting. The sites were selected 
from the publications PC Novice Guide to Web Sites. Web Site Source Book , and The 
Whole Internet . Testing was conducted by the author of this study on an IBM compatible 
computer system configured with the following hardware and software components: 
IBM7CYRIX 200+MHZ Processor, 256k Cache, 16MB Ram; 2.5 GB, IDE Western Digital 
Hard Drive; Trident 9680, PCI SVGA Card, 2MB, MPEG Video Card; Ensoniq PCI, 3D, 32 
BitWave Audio Card with 60W Speakers; USRobotics Sportster External Modem, 33.6 bps, 
V.Fast; Compaq Presario 14" SV Color Monitor; 3 Button Saturn Mouse; 104 Key-Enhanced 
Keyboard; Microsoft Windows 95 Operating System; and Netscape Navigator Browser 3.0. 
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This system was the only one used for the purposes of this study. 

The Data Analysis Phase, Part IV, took place July 1 1th through July 15th. Qualitative 
and quantitative results were analyzed and reported in Chapter V entitled Analysis of Data. 
Descriptive data was reviewed to ascertain the overall strengths and weaknesses of the 
instrument as well as its effectiveness in yielding desired results. Particular attention was 
given to the amount of time required to complete each evaluation and each individual section 
in order to determine whether the process required too little or too much time. The 
quantitative rating system was evaluated to determine whether or not it effectively served its 
intended dual purpose i.e., to provide a meaningful numerical rating for each individual site 
and to aid in distinguishing quality among sites with similar content. Finally, based on the 
results of the post-testing phase, modification and refinement were made to the tool to render 
it functional and effective. 

Several limitations of this study were beyond the control of the researcher. The most 
imposing of these were the vast number of Internet web sites comprising the study’s 
population and the constantly changing Internet environment. It was possible for an 
individual site to have been examined and evaluated on a given date only to be revised, 
updated, or changed completely the next day. 
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CHAPTER V 



ANALYSIS OF DATA 

Testing of the instrument took place as projected from July 1st through July 1 1th. 
During this time, a total of eighteen hours was spent evaluating the thirty pre-selected web 
sites listed below along with the URL, rating, and evaluation time in minutes. Quantitative 
and qualitative data generated during the testing phase exposed both strengths and 
weaknesses of the instrument. 



WEB SITE 


URL 


RATING 


MINUTES 


1. AARPWebPlace 


http://www.aarp.org 


27 


25 


2. American Cancer Society 


http://www.cancer.org 


29 


25 


3. American Civil War 
Homepage 


http://funnelweb.utcc.utk.edu/-- 

hoemann/warweb.html 


27.5 


30 


4. Art Institute of Chicago 


http://www.artic.edu/aic/ 

firstpage.html 


29 


25 


5. Bartlett’s Familiar 
Quotations 


http://www.columbia.edu/acis/ 

bartleby/bartlett 


27 


30 


6. Brittanica Online 


http://www.eb.com 


29 


45 


7. Car and Driver 


http./Zwww.caranddriver.com 


29.5 


30 


8. CIA 


http://www.odci.gov/cia 


30 


40 


9. Consumer Information 
Center 


http://www.gsa.gov/staff/pa/cic 


27.5 


40 


10. Dow Jones 


http://bis.dowjones.com 


27 


40 


1 1 . Emily Dickinson Page 


http://lal.cs.byu.edu/people/black/ 

dickinson.html 


22 


35 
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WEB SITE 


URL 


RATING 


MINUTES 


12. Fodor’s 


http://www.fodors.com 


27 


45 


13. The Geneology Home Page 


http://www.genhomepage.com 


21.5 


30 


14. How Far Is It? 


http://www.indo.com/distance 


18.5 


40 


15. Major League Baseball 


http://www.majorleaguebaseball.com 


24 


55 


16. Mennolink -Mennonite 
Information Center 


http://www.prairienet.org/community/ 

religion/mennonite/menno.html 


25 


25 


17. Merriam-Webster Online 


http://www.m-w.com/dictionary 


29 


35 


1 8. Mutual Funds Home Page 


http./Zwww.fundsinteractive.com 


23 


25 


19. NASA 


http://www.nasa.gov 


29 


35 


20. NBA 


http://www.nba.com 


29 


20 


2 1 . Parenthood Web 


http://www.parenthoodweb.com 


27 


30 


22. PC Webopaedia 


http./Zwww.pcwebopaedia.com 


27 


45 


23. Peterson’s Education Center 


http ://w w w .petersons. com 


29 


45 


24. RxList Drug Index 


http://www.rxlist.com 


23 


35 


25. Social Security Office 


http://www.ssa.gov/SSA_Home.html 


30 


50 


26. Test PREP 


http://www.testprep.com 


25.5 


45 


27. USA Today 


http://www.usatoday.com 


29 


45 


28. Virtual Hospital 


http://vh.radiology.uiowa.edu 


29.5 


60 


29. Weather. Com 


http://www.weather.com 


29 


35 


30. White House 


http://www.whitehouse.gov 


30 


40 




TOTALS 


809.5 


1,105 
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Statistical analysis of the data shed light on the overall functionality of the tool and 
effectiveness of the rating system. Measures of central tendency and dispersion relevant to 
this study were mean, mode, and range. The mean calculated for web site ratings was 27, and 
the mean for the number of minutes required to complete an evaluation was 37. This latter 
figure was broken down even further to determine the amount of time required to complete 
each individual section of the instrument. These results were as follows: Part I - Technical 
Description: 9 minutes; Part II - Multimedia Feature Analysis: 10 minutes; Part III - Content 
Analysis and Evaluation: 12 minutes; and Part IV - User Interface Evaluation: 6 minutes. 

Mode was calculated for both the “Rating” distribution and the “Minute” distribution. 
These figures revealed that the most frequently assigned rating to a web site was 29 and that 
the most frequently occurring time period spent evaluating web sites was 45 minutes. A final 
statistic generated from the “Rating” distribution was that of range calculated at 1 1 .5. 
Although this statistic is not extremely important, it may serve as a type of “alerting signal” 
for the individual reviewer. If in the future, for instance, ranges of web site scores were 
consistently wide, this may indicate that the evaluator should be more attentive to the initial 
web sites being selected for evaluation. 

Of the above-mentioned statistics, the most anticipated and significant was the 
average amount of time required to complete a single evaluation. The reported 37 minutes 
was almost twice that of the original projection estimated to be twenty minutes. After 
examining the time averages for each individual section, it did not appear that these numbers 
were skewed. In fact, these time frames corresponded with the length and amount of detail 
in each section. Part III - Content Analysis and Evaluation was the most extensive section 
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requiring an average of 12 minutes to complete followed by Part II - Multimedia Feature 
Analysis which required 10 minutes. Part I - Technical Description required 9 minutes and 
Part IV - User Interface Evaluation required 6 minutes. The fact that the review process 
lasted longer than projected was merely a moot point since the instrument was successful in 
producing desired results within what may still be considered a reasonable time frame. 

The rating system proved to be an efficient, effective means of representing data 
collected in each part of the instrument. The calculation table located on the cover sheet was 
especially helpful in bringing together all of the individual scores and then generating a final 
composite rating. The system performed extremely well in accomplishing its original two 
goals: to provide a quantitative indicator of quality and to serve as a means of justification 
for qualitative data. 

During the Data Analysis Phase, it became apparent that two entirely new questions 
relevant to the categories “Searching” and “Informative Feedback/Support” would enhance 
Part IV - User Interface. These questions, incorporated into the final version of the 
instrument were: “Is an FAQ section present at the site?” and “Does the search facility 
contain a “Help” feature?”. 

In conclusion, it is important to bear in mind that further implementation of this 
instrument in various library settings will bear different results and that these findings serve 
only as guidelines. There are a myriad of factors beyond the control of the evaluator which 
impact the review process and affect the final outcome. Some of these factors are: type of 
computer hardware and software, particularly the platform and browser; quality of service 
of the Internet Service Provider; type of phone cables in a given area; various skill levels of 
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reviewers; complexity of web sites; size of web sites; and extent of detail at a site. 




CHAPTER VI 



SUMMARY AND CONCLUSIONS 
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As technology continues to grow and the number of web sites continues to increase, 
librarians must establish a system and be armed with the proper tools in order to manage the 
ever-increasing quantity of Internet information and to meet the continuing demand for 
identifying quality sources. Until now, there has been no comprehensive evaluation 
instrument to fulfill this critical need. Consequently, the development of a web site 
evaluation instrument for adult service librarians was executed as part of the Library and 
Information Science Master’s Research Program at Kent State University over a three month 
period from April 15, 1997 to July 15, 1997. 

The objectives of this research project were met by examining the disciplines of 
library and information science, computer graphic design, and linguistics to determine the 
most appropriate, up-to-date evaluation criteria. Ideas and findings of authorities in these 
fields were identified and analyzed to determine applicability to this study. Traditional print 
evaluation criteria and their application to the digital medium, discussed by authors Grassian, 
Caywood, Tillman, Brandt, Smith, Tate, and Alexander, played a major role in the design of 
the tool. These criteria were modified and combined with graphic design criteria and 
linguistic criteria to form a four-part practical tool. 

If in the future this instrument is implemented in the context of an Internet reference 
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and review program, librarians should keep in mind that there are many criteria to be 
considered when determining the quality of an Internet web site. No single element alone 
such as graphics, content, or user interface determines quality regardless of the fact that some 
are more important than others. All must be taken into consideration, evaluated, and placed 
in proper perspective to make a thorough, objective assessment of the value and quality of 
a site. 

The evaluation instrument was designed with this specific aim in mind, i.e., 
affording reviewers the opportunity to examine multimedia, content, and user-interface while 
at the same time providing a quantitative means of rating each respective section. If 
librarians choose to alter the rating system, they may do so by changing the weight of the 
multipliers or by readjusting the rating scale. The tool is flexible and adaptable to any library 
setting. Finally, the instrument was conceived for the sole purpose of evaluating individual 
web sites, not search engines or catalog sites. This model can now serve as a basis for further 
development of instruments that will address the aforementioned Internet sites. 
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If “other,” please explain. 
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1. Is content accessible in 3 mouse clicks or less? 


2. Are the navigational aids present at the top and/or bottom of the pages? 


3. Are navigational aids clearly labeled? 


4. Are pages linked back to the homepage? 


5. Does the use of frames restrict ease of navigation? 
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1 . Is a search feature or search engine present? (If “no,” do not answer 2-6) 


2. Does the search engine index the entire resource? 


3. What type of searching is available at the site? Indicate Below. 

Keyword Exact Match Weighted Other 


4. Are search features available? (If “yes,” indicate below.) 


1 Boolean Operators Truncation Adjacency Operators _JDihcr 


1 5. Is the format display of search results understandable? 


1 6. Can the user manipulate search results? 


7. Does the search facility have a “Help” feature? 
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1. Are types of information, e.g., text, symbols, graphics, etc. 
clearly distinguished from each other? 


2. Does important information stand out on the screens? 


3. Is it clear where the user enters information on the screens? 


4. Is it clear in what format information is to be entered? 


5. Are light colors displayed on a dark background? 
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7. Does the use of certain colors discourage viewing over long time periods? 
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1 I. Do instructions and prompts clearly indicate what to do? 


2. Are instructions consistently worded throughout the site? 


3. Are status messages present to indicate what the system is doing or has 
done? 


4. Does the system inform the user if errors occur? 


5. Does the system allow the user to correct errors? 


6. Is it made clear to the user how to correct errors? 


7. Does the site contain large print to aid those with visual disabilities? 


8. Does the site contain an FAQ section? 


9. Is a “Help” feature present? 


10. Is it clear how to access and exit the help facility? 


1 1. When using the “Help” feature, does the system explain actions in the 
context of what the user is currently doing? 


12. When using “Help,” can the user find information efficiently, avoiding 
unnecessary information? 
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APPENDIX B 



USER’S GUIDE 



This instrument has been developed to meet the adult reference librarians’s current, 
critical need for a comprehensive World Wide Web site evaluation and reviewing tool. The 
accompanying User’s Guide provides important information and guidance for supervisors 
overseeing the evaluation process and for evaluators and reviewers themselves. Topics 
addressed in the User’s Guide include; instruction for proper usage, definition of terms, and 
integration of the instrument into the selection process. 

Experienced Internet librarians, as well as new reference librarians, will find this 
evaluation instrument effective and easy to use in identifying quality web sites (search 
engines and catalogs not included). The instrument addresses content and design elements 
using the methods of feature analysis and content analysis. Evaluation criteria are presented 
in the form of checklists and questionnaires which enable the reviewer to complete the forms 
in a systematic, methodical manner. A tickmark system, indicating the presence or absence 
of these criteria, is the means by which evidence is gathered for the justification stage of the 
final decision-making process. 

The instrument is composed of four main parts and is to be implemented in the 
following order: Part I - Technical Description; Part II - Multimedia Feature Analysis; Part 
III - Content Analysis; and Part IV - User Interface Evaluation. The average amount of time 
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required to complete a review session is 37 minutes. This time frame will vary depending 
upon various factors such as the complexity and detail of the web site. It is advisable to 
conduct a complete evaluation in one session in order to gain a more thorough understanding 
of the site. Before begiiming Part I, evaluators should direct their attention to Page 1, the 
Cover Sheet or “Recommendation Form.” Instructions for completing this form and the four 
major parts of the evaluation instrument are included in the pages that follow. 
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RECOMMENDATION FORM 



Page One of the instrument, the “Recommendation Form,” serves the dual purpose 
of compilation sheet and approval form. Important descriptive and quantitative data, to be 
recorded in sections A, B, and C will provide reference librarians and department supervisors 
with an overview of the Internet site. Section A, to be filled in at the onset of a review 
session, includes a place for the reviewer’s name, date, start time, and end time. Section B 
contains the rating table and a place for commentary regarding the approval or rejection of 
the web site as a reference source. The rating table allows the reviewer to assign numerical 
value to a site. Its purpose is best served when comparing and selecting sites similar in 
purpose and scope. Upon completing Parts II, III, and IV, the reviewer assigns a rating from 
one to five based on the scale below and transfers the number to the respective lines, la, lb, 
or Ic, of Page One. 

5 - Excellent. The site contains all of the features and meets all of the criteria 
exemplifying superior quality. 

4 - Above Average. The site contains almost all of the features and meets 
almost all of the criteria exemplifying above average quality. 

3 - Average. The site contains enough features and meets a sufficient number 
of criteria that it is considered satisfactory. 

2 - Below Average. The site contains an insufficient number of features and 
does not meet enough of the criteria to be considered satisfactory. 

1 - Failing. The site is not worth considering for this library. 
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The rating system employs the weighted multipliers 1, 2, and 3. As in the evaluation 
of any reference source, content is the primary consideration and is, therefore, weighted by 
the multiplier 3. The second most important feature, the use of multimedia, is critical to the 
conveyance of content and is weighted by the multiplier 2. The third most important area, 
User-Interface Design, is assigned the multiplier 1 . If this rating system does not meet the 
needs of a library, it cein be deleted from the review process entirely or tailored to a specific 
situation. 

The final decision regarding the accepteince or rejection of a web site as a reference 
source should be made by weighing the quantitative and qualitative results of the evaluation 
against those criteria stated in the library’s selection policy. Once the evaluator has indicated 
his conclusion in Section B, the complete set of forms should be submitted to the head of 
reference or a supervisor for official approval. (See Section C) It is suggested that the 
department head retain the originals and return a copy to the reference department. All 
reviews should be placed together in a file or binder designated “Web Site Reviews ” and 
kept on heind at the main reference desk. The purpose of this collection is not only to serve 
as a handy ready reference tool for librarieins, but also to serve as ein official log of the 
library’s own “bookmarked” Internet sites. 

Finally, libraries should strongly consider incorporating this web site evaluation 
instrument into their selection policy. World Wide Web sites must be treated as would any 
other media. Questions regarding this issue that need to addressed include: Who is 
responsible for conducting reviews? How often will reviews be conducted? How many sites 
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does the library intend to review? From what sources will the library choose potential 
Internet sites? and How often will sites be reevaluated and / or deselected? - a key issue 
given the highly fluid state of the Internet. 

These basic questions comprise the fundamental principles of an Internet Collection 
Development Policy upon which librarians will want to expand. Whether this means 
developing a separate policy, or revising and updating an existing one, this action is an 
essential step in the Selection and Acquisition Process. 
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PART I - TECHNICAL DESCRIPTION 



Part I, the Technical Description, is a fill-in form which serves as a recording device 
for descriptive data. For the most part, this information can be located on the home page of 
a web site. Definitions, along with a brief explanation of the terms appearing in Part I, are 
listed below in alphabetical order. 



Approximate No. Of Inward Links - An estimate of the number of links 
leading to various types of files such as audio, video, graphics, text, etc., 
within the site. 

Approximate No. Of Outward Links - An estimate of the number of links 
leading to various types of files such as audio, video, graphics, text, etc., 
outside the site. 

Author - Person(s) responsible for the intellectual content of the web site, 
usually indicated in the footer area of the home page. 

Browser Type and Level Required - Type of web browser and version 
number required to display the web site. Examples include: Internet 
Explorer, Netscape Navigator, and Spyglass Mosaic. 

Cost - Free or fee-based. 

Date of Last Revision - When the site was last changed or revised, usually 
indicated at the bottom of the home page, or sometimes at the bottom of 
every page. 

Date of Publieation - The date the intellectual content was originally 
published, usually found at the bottom of the home page, or sometimes at the 
bottom of each individual page. 

Date of Site Mounting - When the information was placed on the Internet, 
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usually indicated on the footer of the home page. 

Editor / Agent - Person(s) who have made other significant contributions to 
the intellectual content of the work. 

Genre - Type of Source. Select from the list below, or indicate “other” on the 
line provided and explain. 

Academic Institution home pages 
Advertising Site for a Product 
City Site 

Collection of Links focused on a special subject or theme 
Comedic Pages 
Company Web Site 
Directory ( phone, map, etc.,) 

Electronic Journal or Magazine 
Federal Government Pages 
News Source(s) 

Organization and / or Association Pages 
Personal Site 
University-based Project 

Host Institution - Agency responsible for making the object available. 
Examples are: Company, government, university, etc., usually indicated in 
the header or footer areas. 

Site Name - Official title of site, usually found on the header of the home 
page. 

URL - Uniform Resource Locator, the unique address of the web site. 

Language(s) of Site - Primary language(s) such as English, French, Spanish, 
etc., used at the site. 

Language(s) of Site Content - Language(s) of the intellectual content of the 
site. 

Other - Any other person acknowledged at the site for having made an 
intellectual contribution to the work. 

Plug-Ins Required - Hardware or software modules that enable browsers to 
display specific audio or video features, usually indicated on the home page. 





PART II - MULTIMEDIA FEATURE ANALYSIS 



The complex interactive environment of the Internet hosts a variety of media through 

which information is communicated. In Part III, Multimedia Feature Analysis, the evaluator 

is required to maneuver throughout the site to determine the presence or absence of specific 

elements such as; graphics, audio, video, and animation. The use of criteria checklists and 

sets of questions will aid the evaluator in determining whether the multimedia design 

enhances or distracts from the content. The terms appearing in Part II are defined and 

explained below in alphabetical order. 

Animation - Movement created by displaying a series of pictures or frames. 

Continuous Play - Nonstop, continual from the onset of initialization. 

Decorative - Providing visual appeal and emphasis. 

Digitized Image - A graphic or image that has been manipulated with a 
computer. The purpose of the image may be representational, organizational, 
explanative, or decorative. The image may appear in black and white or 
color. 

Explanative - Showing how processes work. 

External - Outside the web site. 

Icon - A graphical representation of an object or program. Icons can be 
classified as decorative, representational, organizational, or explanative. Web 
sites may contain all types of icons, some types of icons, or none at all. They 
may appear in color or black and white. 

Imagemap - A clickable image that contains a link to a different location 
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either within the web site or outside the web site. The surrounding area on 
which the user clicks is termed the “hot spot.” Imagemaps may appear in 
black and white or color and be classified as decorative, representational, 
organizational, or explanative. 

Inline - Internal to the web site. 

Interactive Audio - Sound generated by the user, usually produced by 
clicking on an icon. 

Music - Vocal or instrumental sounds having rhythm, melody, and harmony. 
Natural Sound - Sounds taken from nature. 

Organizational - Depicting relationships among items mentioned in the text. 

Representational - Containing items mentioned in the text. 

Synthesized Sound - Computer generated sounds that are neither produced 
by humans nor taken from nature. 

Synthesized Speech - Human speech generated by a computer. 

User-activated Animation - Motion that is initialized by the user, usually by 
clicking on an icon. 

Video - Visual portion of a movie clip or film that is prepared for viewing on 
a computer monitor. 

Video File - Video and audio combined. 
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PART III - CONTENT ANALYSIS AND EVALUATION 



Having made an initial ^praisal of the web site by recording the technical data and 
examining the multimedia features, the evaluator should now be prepared to analyze the 
intellectual content of the site. Content is the primary consideration in evaluating any 
reference source. The numerous elements of content are quite complex and require 
considerable deliberation on the part of the evaluator. The elements appearing in Part III are 
defined and described below in alphabetical order. 



Accuracy of Intellectual Content - The information presented at the web 
site corresponds to known facts. Check for the presence of a bibliography or 
links to sources cited in the text. Look for names of individuals or sources 
from which nonpublished data was obtained. For a research site, check for 
an explanation of research methods. 

Accuracy of Document (Linguistics) - Correct spelling of words, acronyms, 
and abbreviations, proper syntax, proper semantics, correct usage of 
alphanumeric information. Consult Prentice Hall Handbook for Writers or 
a similar reference source to verify accuracy. 

Authority - Characteristics pertaining to the quality of a site. Determine 
whether the site has a reputable organization or expert behind it. Look for 
standardized names. Check headers and footers to see who is producing or 
sponsoring the site. The URL may provide clues as to the authority of the 
source. For example, a tilde usually indicates a personal web directory. 
Other indicators may be the following URL domains: 

“.edu” - domain of a server representing an education institution 
“.gov” - domain of a server representing a government 
“.com” - domain of a server representing a commercial business 
“.net” - domain of a server representing a network 
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“.org” - domain of a server representing an organization 

Note that authority may or may not be related to the author or sponsoring 
agency. 

Coverage - Subject areas presented, time periods covered, geographic areas 
covered. Coverage may be evaluated in detail by comparing the web site to 
other sources, such as print sources on the same topic. 

Currency - Timeliness of information. A web site may be static or updated 
regularly. Consult the home page or end page for a statement regarding 
currency. 

Intended Audience - The users at whom the web site is directed. Check for 
a description of the intended audience in the mission statement or 
introduction. Language and writing style may also provide clues as to the 
intended audience. 

Links - An element in an electronic document that connects to another place 
in the same document (inward link) or to an entirely different document 
(outward link), also referred to as a hyperlink. Links should serve as 
screening and pointing instruments for users. Links greatly affect the logical 
layout of a web site and should be thoroughly evaluated during the review 
process. 

Objectivity - The use of balance in the presentation of controversial issues. 
Determine whether the information is fact, opinion, or propaganda. Check 
for the presence of emotion-rousing words and bias. Examine the URL for 
clues as to the origin of the site. 

Organization - Arrangement, structure of a web site. Sites may be arranged 
according to traditional schemes such as: alphabetical, numerical, 
chronological, or geographical, or according to an organizational structure 
such as academic departments or corporate hierarchy, by subject categories, 
or by some other organizational scheme. How well a web site is arranged 
will impact how easy it is to use. 

Purpose - Mission Statement. Look for this statement on the home page. 
Some sites may or may not contain a mission statement. If they do not, their 
purpose may be implied. 

Relation to other Works - Compatibility to other works. Examine other 
sites, as well as print counterparts, similar in scope and purpose. Determine 




PART IV - USER INTERFACE EVALUATION 



User-Interface Design is the area in which criteria for Internet sources differ most 
from other sources, particularly print sources. Never before has there existed such an 
intensely interactive multimedia medium. Issues critical to this area are: Navigation; Visual 
Clarity; Searching; and Informative Feedback and Support. A complete understanding of the 
importance of these elements, along with their rigorous evaluation and review, are essential 
in assessing the quality of a web site. A reference source in which information is poorly 
indexed and organized is seldom used; whereas, a reference source in which information is 
easy to find and readily available is consulted frequently and considered a superior tool. 
Definitions of the terms found in Part IV are listed below. 



Informative Feedback - Information that informs the user what has been 
done or is to be done. Due to the highly interactive medium of the World 
Wide Web, it is crucial to examine this type of feedback and its value at the 
web site. 

Navigation - Maneuverability within a web site. Navigational ease is critical 
in obtaining information at a site. Devices should be present at the top and 
/ or bottom of each page, regardless of the web site size or complexity. 

Search Facility - A program that allows the user to query the site. Some 
sites will have search facilities and some will not. Performing several trial 
searches will help to determine the engine’s capabilities. 

Visual Clarity - The clearness by which symbols indicate their functions and 
relationships. 
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1. Robert Farmighetti, ed., The World Almanac and Book of Facts 1997. 
(Mahwah: K-III Reference Corporation, 1997), p. 567. 

2. James Rettig, Putting the Squeeze on the Information Firehose: The Need for 
Neteditors and Netreviewers [article on-line]; available from http://www.swem.wm.edu/ 
firehose.htm; Internet; accessed 29 April 1997. 

3. Louis B. Rosenfeld, “Guides, Clearinghouses, and Value-Added Repackaging: 
Some Thoughts on How Librarians Can Improve the Internet.” References Services 
Review. Winter, (1994): 11-15. 

4. Matthew Ciolek, The Six Quests for the Electronic Grail: Current Approaches 
to Information Quality in WWW Resources [article on-line] available from http://coombs 
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